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ABow RACT 


The purpose of this thesis is to determine if format 
variation has any affect on speed and accuracy of data 
entry for the Marine Integrated Fire and Air Support System 
(MIFASS). 

Independent variables for the test are message formats, 
message types, and subjects. Subject learning effects and 
experimenter differences are controlled by use of a pilot 
Study. Subjects are included as a treatment in order to 
estimate their effects. 

The experiment is designed to Support both parametric 
and nonparametric analyses. A 4x5x8 randomized complete 
block factorial with two replications is used to analyse 
the effects of format variation. Friedman's Two-Way ANOV 
by Ranks is used to test subjective rankings of the formats 
as concerns ease of use and operational SU ital acy. 

It is concluded that format variations do affect speed 
of data entry but not accuracy. Format 3 appears to be 


Superior. Recommendations for further study are given. 
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LoL EMEN TL OF THE PROBLEM 


Awe tHe PROBLEM IN GENERAL 

Them primagweeuner on Of a Fire Support Coordination 
Center (FSCC) at any level of command is to provide the 
commander with the means for planning and coordinating 
available fire support. Specific responsibilities of a 
FSCC include fire support planning, designation of targets 
and asSignment of priorities, allocation of fire Support 
resources, and coordination of supporting arms with each 
other and the supported force. At present, all these 
responsibilities and functions are carried out manually 
through the use of radio and telephone communications, 
personal liaison, and an immense amount of paperwork. An 
inordinate amount of valuable time is lost in the FSCC 
during normal operations because of the paperwork and 
liaison required to accomplish its mission. In an attempt 
to eliminate this bottleneck and more effectively use the 
Skill and training of the personnel who operate the FSCC, 
the Marine Corps, in conjunction with Hughes Aircraft 
Company, is developing an automated system which will 
materially reduce the paperwork, list keeping, and manual 
labor presently required in a FSCC. The system will be 
called the Marine Integrated Fire and Air Support System 


(MIFASS). 





B. OVERVIEW OF THE AUTOMATED FSCC 

The concept for MIFASS operations offers extensive 
improvement in tactical effectiveness through automation of 
fire request planning, fire support coordination, fire and 
air support planning, conflict prediction, and the consoli- 
dation of many other common functions at the MIFASS centers. 
The automated system will utilize various configurations of 
the hardware listed in Table I at designated levels of the 
Sretatronaletactacal vnit. ~Subunits operating under tacti— 
cal control of the MIFASS centers will input intelligence 
data, fire support requests, med-evac requests, damage 
assessment reports, etc., to the MIFASS computer. MIFASS 
software will provide rapid response to many on-line users 
by interleaving the processing tasks and overlapping the 
input/output (1/0) demands with processing. Through 
generalized data management software, the system will permit 
definition of new messages, files, reports, and displays in 
the field without reprogramming. The application programs 
envisioned will include computer assistance and algorithm 
development for major requisite functions such as fire 
mission analysis, target data listing, air support control, 
technical fire control, conflict detection, troop safety, 
mission scheduling, resource allocation, etc. Since each 
FSCC will have access to the memory bank and coordinating 
agencies will be able to view the same data simultaneously 
on their own output units, the time needed to perform the 
required functions for all subunits will be materially 


decreased thereby increasing the effectiveness of the 
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Overall system. Due to the probability that several agencies 
will be viewing the same data simultaneously and in the 
interests of simplification in training and programming, 

it is evident that a simplified and uniform data I/O pro- 


cedure will be required. 


TABLE I 


HARDWARE REQUIRED FOR A FIELDED MIFASS 


Each ESCG will employ vaspeciftic configuration of some or 
all of these units. 
Digital Message Entry Device (DMED) 
Portable Printer 
Remote Terminal Device (RTD) 
Cathode Ray Tube (CRT) Display 
Small Auxiliary Memory 
Hard Copy Printer 
Keyboard Entry Panel 
Communications Net 
Computer with Internal Memory 16-131K 
Auxiliary Mass Memory 500K-8M Characters 
Magnetic Tape Units 
Printers 
Plotters 
Large Screen Displays 
Textual/Tabular Displays 


Graphic Displays 


Jed 





CVO ALA ENiERy PROCEDURES 

Proper data I/O procedures are crucial to the function- 
ing of an automated system. One phase of these procedures 
which presents a potential bottleneck to the system is the 
format used when entering data or outputting it on a grid 
display or copier for viewing by interested parties. Ifa 
uniform format is not used by everyone, users will encounter 
problems in data retrieval and in interpretation of data 
entered by other on-line units. The requirement for a 
format (or formats) understood by all which is fast, accu- 
rate, and convenient is obvious. Determining the feasi- 
bility of such a format was the overall objective of this 


experiment. 


pees’ URP OSE OF THE TEST 

As one approach to determining the feasibility of a uni- 
form format for use in the system, this test was designed 
to evaluate differences in operator data entry speed and 
accuracy resulting from the use of alternative message 
formats when used with differing types of I/O data. The 
convenience and operational suitability of the formats 


tested was to be determined through operator surveys. 


Peete OCHESGE oO LHAL WERE LESTED 

Hou? There is no significant difference in the speed of 
data entry due to the use of different message formats. 

Ho! There is no significant difference in the accuracy 
of data entry due to the use of different message formats. 


lee 





Ho3° There.is no significant difference in the speed of 


data entry due to the use of different message types. 
Hoa? There is no significant difference in the accuracy 
of data entry due to the use of different message types. 


Hos? There is no significant difference in the speed of 


data entry due to the subjects used for the test. 


Hoe? There is no significant difference in the accuracy 


of data entry due to the subjects used for the test. 


Ho: There is no Significant interaction between message 


type and format for speed of data entry. 
Hy 9? 


EyoomaincrOmtat, tOr accuracy OF data entry. 


There iS no significant interaction between message 


Ho9: There 1s no significant interaction between message 
format and subject for speed of data entry. 


H There is no Significant interaction between 


010° 
message format and subject for accuracy of data entry. 

Holl There is no significant interaction between 
message type and subject for speed of data entry. 

Ho12: There is no significant interaction between 
message type and subject for accuracy of data entry. 

A613? There are no significant interactions among 
message type, message format, or subject for speed of data 
entry. 

Hoa? There are no Significant interactions among 
message type, message format, or subject for accuracy of 
data entry . 


H The subjects are indifferent to the message for- 


015° 


mats concerning ease of use. 
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H The subjects are indifferent to the message for- 


O16" 


mats concerning operational suitability. 


F. ASSUMPTIONS 
1. Message Types 
The message types selected for this experiment were 
assumed representative of the total population of messages 
envisioned for use in an operational MIFASS. The messages 
chosen varied in length from 62 to 573 characters which were 
a mixture of numeric and alpha-numeric; therefore, they 
covered a wide span of the total population. 
2. smioyects 
The subjects used for this test were assumed repre- 
sentative of the total population of personnel available to 
operate a fielded system. This assumption might be ques- 
tioned in view of the experience and proficiency gained by 
the subjects during the preceding several months of system 
development. However, circumstances dictated that these 
particular subjects be used, and, in any case, it is a 
certainty that personnel assigned to operate a fielded 
system will undergo several weeks of intensive training 
before being permitted to begin actual operational duties. 
Thus, this assumption was considered justified. 
3. Message Format Effects 
Effects of different message formats on the speed 
and accuracy of data entry under test conditions were con- 
sidered typical (ignoring stress, fatigue, etc.) of the 


effects which will be prevalent during actual operations. 
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If it is subsequently determined that any conditions make 
this assumption invalid, this experiment can be repeated 
under those conditions during later phases of MIFASS 


development. 


G. GENERAL APPROACH 
1. The Test 
Four test runs were made, each of which consisted of 
a group of subjects entering data from five message types 
into a specific blank message format which was presented on 
a grid display screen. Each subject entered data twice for 
each message type for a total of 10 trials per run per 
Subject. Speed and accuracy measures were taken for each 
trial. Upon completion of each run, the subjects each com- 
pleted a subjective evaluation form concerning the ease of 
use and operational suitability of that format. 
Zeneca SS 
A randomized block factorial analysis of variance 
was used to test the first 14 hypotheses. The two subjec- 
tive hypotheses were tested using nonparametric test 
procedures. 
3. Remarks 
It was not the objective of this experiment to choose 
a single format for use in a fielded system but to test the 
feasibility of such a format. However, if one of the for- 
mats tested were to be found faster and more accurate than 


the others under all test conditions, it might be assumed 


i 





that that format would be a good possibility for adoption 


as a universal format for the MIFASS. 
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Pe oelec. LON OF DESIGN 


A. DEPENDENT VARIABLES 
Tee peeo OL Dataventry 
The time, Ts, required to enter each stimulus 
message into a blank message format was collected for each 
subject during each experimental run. This response was 
then normalized by dividing each T; by the total number of 


od 


keying actions required for that message format including 
any header data or spaces over which the operator had to 
Space or tab manually. Thus, the dependent variable speed 
in this test was the normalized data entry time. 
23 Accuracy Os Wess Nae ay 

The accuracy of data entry was defined in terms of 
data elements which are entire sets of one or more charac-— 
ters to be entered into a single message field. The number, 
A., of data elements that had been entered incorrectly 
was collected for each message entered by each subject dur- 
org Bele experimental run. Any error such as single or 
multiple typographical errors within a data element, an 
omitted data element, or one data element substituted for 
another was counted. This response was normalized by 
dividing A. by the total number of data elements that were 
entered in the ge message. Thus, the dependent variable 
aceuracy 1h he eee was the normalized data entry accuracy 


=sCOLEe. 


aly 





3. Subjective Rankings 


To forestall the possibility that individual memories 
[citrine ctmtieeranking OF the formats if the subjective 
Opinions were to be gathered after all the formats had been 
presented, it was considered necessary to gather a separate 
Opinion from each subject for each format immediately after 
it had been tested. A graduated interval scale was designed 
for each subjective dependent variable. Each scale con- 
sisted of 20 intervals with the endpoints and some inter- 
Mediate intervals marked with verbal descriptions for ease 
in marking as shown in Fig. 1. These scales were then used 
for gathering the individual opinions. 

a. Ease of Use 

The graduated scale used by each subject to give 
his opinion of the ease of use of each format was marked 
from 0 to 20 where 0 coincided with the rating "Easy to Use" 
and 20 matched "“Unusable." 

b. Operational Suitability 

The opinions for this subjective dependent 
variable were marked on a graduated scale where 0 coincided 


with "Very Suitable" and 20 with "Very Unsuitable." 


B. INDEPENDENT VARIABLES 
1. Message Formats 
The independent variable of primary interest was 
message format. There were four different formats tested 
in this experiment. Appendix A presents the actual formats 


tested for each message type. 


rs 





SUBJECT REACTION FORM 


Sub ect Format 


Experimenter Denes 


Please indicate your reaction to the format you have just 
completed testing by placing a checkmark at the appropriate 


place on the line below. 


l. How easy or difficult was it to use this format? 


0 5 10 ILS 20 
Easy Convenient Useable May be Unuseable 


to Use Useable 


2. How would you rate this format as far as suitability 


for use in an operational MIFASS? 


0 5 10 15 20 
re ee eS ee ee eee en een ee ee a en en a el | 
Very Suitable Possibly Unsuitable Very 
Suitable Suitable Unsuitable 

Figure 1. Subject Reaction Form. This form was used to 


gather data necessary to the nonparametric 
analyses required. 


a9 





a. Linear Format 
In this format, each field name in the message 
was immediately followed, on the same line, by a space OG 
Spaces in which the pertinent message data for that field 
was to be entered. One field followed the previous field 
on the same line without spacing between fields until that 
line was filled. The succeeding fields were positioned in 
lines as stated until the entire message had been completed. 
i. welock Format 
The block format had the field names in a linear 
sequence with the message data space or spaces immediately 
below the field name on the next line of the display field. 
The field names were separated sufficiently to preclude 
information overlap in the message data spaces below. No 
space appeared between the line of field names and the line 
containing related message data. A double space appeared 
between each adjacent pair of lines. This form was followed 
until enough line pairs appeared to complete the entire 
message. 
c. Columnar/Separated Format 
For this format, all field names appeared in a 
Single column on the left side of the display area. The 
Space or spaces for the pertinent message data for each 
field appeared in a second column indented from the left 
margin of the display area, not from the end of the field 
name. Thus, the message data column appeared as a distinct 


column with a uniform margin. 
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dad. Columnar/Run-On Format 
This format had the field names in a column as 
above, but the space or spaces for message data appeared 
without spacing to the immediate right of the field name. 
Instead of a single column of field names-message data as 
used above, three columns were used where space on the 
display area existed; otherwise, two columns were used. 
@emeeeosage types 

An independent variable of secondary interest was 
message type. As previously discussed, the five types 
selected for this experiment were assumed to be representa- 
tive of the total population of messages to be found in a 
fielded system. Data pertinent to these message types is 
shown in Table II. The actual messages used form Appen- 


dix B. 


LABiLE Tt 


DATA ON MESSAGE TYPES 


Msg. Msg. Ms None Number of Number of 
Type Name ok Characters Fields 
i CHKFIM Check Fire Command 62 6 
2 EOMBDM End of Mission Battle 96 10 
Damage Assessment 
8 TGTLST Target List Input Iv it 14 
4 CFEIGR Gall for Fire by 266 INE: 
Target Number 
= FORDRM Fire Order Universal SS 54 


Zl 





Silents 


The other independent variable of secondary interest 
was the subjects participating in the test. As discussed, 
these subjects were assumed to be representative of the 
total population of potential system operators. However, 
their previous training and experience might have affected 
their performance and might have introduced bias into the 
experiment. The inclusion of subjects as treatments per- 
mitted an analysis of their effects on the dependent vari- 
ables to be made. Eight subjects were used. 

4. Replications 

The independent variable of tertiary interest was 
replication. The inclusion of this variable was principally 
to provide a check to see if the experiment was conducted 


consistently. 


C. CONTROLLED VARIABLES 

There were several nuisance variables which had the 
potential to bias the outcome of the experiment. Subjects 
were such a potential source of bias and their effects were 
controlled by including them as a treatment in the experi- 
mental design. Other nuisance variables could not be exper- 
imentally controlled as were subjects but had to be con- 
trolled by other means. 

Y. Abbreviations and Acronyms 

This source of bias was controlled both experimentally 


and statistically. The use of abbreviations and acronyms in 
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all message types is not necessarily proportional to the 
length of the message; therefore, the normalization of the 
dependent variables as discussed in Para. II.A was also 
utilized to statistically control this effect by keeping 
the variation more nearly constant over the normalized data 
base. Standard abbreviations and acronyms were used for 
all message types for all formats. This assumption held 
these effects constant. 
2 eee Pe rIlcnren sp ELEHoLrences 

Two experimental controls were used to minimize 
effects due to potential experimenter differences. First, 
all experimenters were trained to administer the experiment 
in a standard fashion thus minimizing effects due to differ- 
ent testing procedures. Second, each experimenter presented 
each format twice and administered no more than one test 
run to any given subject. 

3. Subject Learning Effects 

To minimize learning effects, two experimental con- 
trols were used: training and randomization of presentation 
of formats. All subjects completed the Marine Tactical 
Command and Control System (MTACCS) Test Bed Operations 
Training Course prior to being administered the test. Such 
training ensured familiarity with all five message types 
and all four formats. Thus, when proceeding from a test 
run with one format to the next run with another, a subject's 
learning effects were minimized. Remaining effects which 


could possibly introduce bias were reduced by randomizing 
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the order of presentation of the formats and message types 
to the subjects. 
4, Stimulus Presentation 
a. Message Input Stimuli 
The stimulus messages were presented in hard 
copy form only. Radio or voice presentation was deemed too 
difficult to control with the facilities available for the 
test. See Appendix B. 
pee Ormate a cam i 
The formats used for this test were presented 
to the subject on the grid display screen. One format was 
presented at a time in the order shown in Table III. The 
random ordering of the presentation of each format minimized 
any possible effects from subject learning that could have 
occurred if the formats had been presented to all the sub- 


jects in the same order. 


Dew  oLATISTICAL DESIGN 

Since it was desired to secure data from every combin- 
ation of the independent variables, a randomized complete 
block factorial design, 4%x5x8 with two replications, was 
selected for this experiment. This design required 320 
data points with which to test the hypotheses and provided 
an extremely high power for the test. Table III presents 


the test schedule required to support this design. 
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TABLE er. 1 aeiCont.) 


5 2 to) 2Sae ces ail a, 
2 2 2 ec 
nee oes UCC 
2 i: | ST 
A 2) ee ee 
Run enews 22 2. 1 2 
2 3) 20 2 Cee 
iene i =e 4 4 4 
i Oe es 
Siete i) da 
Boy erie aye 2G) Eas ees 


ie Sosunp eons 
a. Model 


The mathematical model assumed for this design 


was a fixed factor model as developed in Ostle [Ref. 1]: 


eal = Oe ee ace yee eee oa 
T (ABC) oP! 5 5K 
where Xi kl = observation ijkl 
u = mean effect (constant) 
R; = effect of replicate 1, 1=1,2 
Ps = effect of message format j, j=1,2,3,4 
By. = effect of message type k, k=1,2,3,4,5 
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Cy =—EomeocteOr Suayect 1, l=1,2,...,8 
UE) etc. are interaction terms 

e.. = random error for observation ijkl 
agi eal 


It was assumed that e. was distributed normally with mean 


J ycal 
zero and variance oe where e 1S homoschedastic for all i,j, 
k, and 1. It was further assumed that there were no repli- 
cation by treatment interactions and that the observations 
were independently distributed multivariate normal. 
b. Treatment Effects 

As discussed, all factors in this model were 
assumed to be fixed. There is no doubt of this for the 
formats and message types; however, the subject sample 
might be argued to be random rather than fixed. Consider- 
ing the experience gained by all the subjects through 
several months familiarization with the test bed and its 
equipment and their overall level of past Marine Corps 
operational experience and training, it was decided that 
generalization of the conclusions made from this test over 
the entire population of possible field operators would not 
be totally valid. Therefore, of necessity, this factor had 
to be considered fixed, also. 

c. Effects of Nonconformity to Assumptions 

In general, the consequences are not serious 
when the assumptions made in connection with the analysis 
of variance are not strictly satisfied. For example, minor 
deviations from normality and/or some degree of heterosche- 


dasticity will have little effect on the usual tests and 


a) 





the resulting inferences. Reference 2 discusses this aspect 
of experimental design thoroughly. 
yo SNE INS) LEAS 

To determine the proper sample size to use for this 
test, one has to use ideas developed by Tang [3] and used 
by Pearson and Hartley [4] in the development of their 
power function curves. Using these developments, one cal- 
culates the sample size required to yield a specified power 


for the desired test utilizing this equation: 





5 ple 
nial 
k 
d = 
Oi. sial 
where d¢ = power function 
D; = difference between the effect of the yeh 
treatment and the mean effect of all treatments 
k = number of treatments (i.e., number of levels 
of the treatment in question 
Oo = standard deviation of errors 
n = number of observations required for each 


treatment level 


Since, for this experiment, o and D; were not known, a modi- 
fication to this equation was made to circumvent these 
problems. It was assumed that a maximum detectable treat- 
ment difference of one o about the mean would be sufficient 


for this experiment. It can be shown that the minimum AIS 


Seeurs when two Of the treatment differences are - a and 
2 and the remaining differences are 0 [Ref. 7]. Then 
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2 2 2 
BN I Co. =o 
r,2 = ( | +(§) te Ole, uae oO = 5 


Hence, if the maximum detectable difference is other than 
that chosen, the power of the test will increase. Now, the 


above equation for $9 becomes 





This form was much simpler to utilize and was no longer 
sensitive to errors in estimating o. Using this power 
function with the proper degrees of freedom and a = 0.05, 
the available power can be determined for various sample 
sizes from the curves produced by Pearson and Hartley [4]. 
As shown in Table IV, a power of approximately 0.90 was 
attainable for the test of format differences with as few 
as three subjects, each subject using two samples of each 
message type for each format; whereas, four subjects were 
required to get a Similar power for the test of message 
types. Since there were 40 observations for each subject, 
effects of subject differences could be tested with 
extremely high power for almost any number of subjects. 
However, as the use of eight subjects did not add mater- 
1ally to the cost or time required but did enhance the 


power of any test desired, eight subjects were used. 
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TABLE IV 


AVAILABLE POWERS FOR THE TEST 


Power 


Degrees of 


No. Of 
Slog 


Obs ./Subj./ 
Cell 


Preeaom Lor 
Error Term 


30 


259 


Lo 


Is, 


InES 


1 


Amex) 


32) 


118 


39 


78 


i Bey, 


Format 


09.9.9 


0. 999 


Diego 


) Me ie, 


Om 2 


Oi3'99 


Orcs 


OR 


Ore 0 


Or £0 


Grados 


0.900 





tiie SOGtstrcs AND TRAINING 


hw SUBJECTS 

The eight subjects used for this test were Marine Corps 
officer and enlisted personnel assigned to the MTACCS Test 
Bed which has direct responsibility for developing MIFASS. 
Names of 22 eligible officers and men formed a pool from 
which eight were drawn randomly to act as subjects. Pre- 
requisites for eligibility were previous training or 
experience in fire support coordination, artillery coordi- 
nation, target data analysis, and test bed operations. A 
capability of an extremely high or low typing speed rendered 
potential subjects ineligible as it was desired to have 
typing speed on a level representative of personnel who will 
serve in a fielded MIFASS. Each subject had completed the 
MTACCS Test Bed Operations Course which covered those 
subjects listed in Table V. Prior to their first test run, 


the subjects were thoroughly briefed on the experiment. 


Bee EXPERIMENTERS 

The four experimenters necessary for this test were 
selected similarly to the Sanseses except that all test bed 
personnel, including the civilian contractor employees, 
were considered eligible for duty as experimenters. Avail- 
ability was the prime consideration in experimenter selec- 
tion. Those selected were familiar with MTACCS Test Bed 


operations through the above mentioned course and were all 
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TABLE V 


SUBJECTS COVERED IN THE MTACCS 
TEST BED OPERATIONS COURSE 


MUnees Concent 
Role of the Test Bed 
MIFASS TSM Concept 
Concept of Test Bed Operations 


Display and Entry Devices (I/0) 
(Included practical exercises) 


Marine Corps personnel. Additional training included 
experiment familiarization sessions conducted by the test 
administrator, practice sessions, and a pilot study as 
discussed in Para. IV.A. Those selected also attended the 


Subject briefings. 


C. HARDWARE AND SOFTWARE REQUIREMENTS 
1. Hardware 
The hardware listed in Table VI was utilized for 
the performance of this experiment. 
2. software 
The software used for this experiment is discussed 
PienmOugiivwineherto 85 cimee 1t iS not in the public demain 
nor pertinent to a general discussion of the test, a listing 


nS OMmlerea siete, 
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TABLE VI 


HARDWARE REQUIRED FOR THIS EXPERIMENT 


Unit Model Quantity 
CDC Computer System 3300 1 
CDC Grid Display System 243-1 1 
Call Director 600 2 
Voice Recorder Reh 720 Jt 
Time Code Generator hea Om 1 


D. TEST EQUIPMENT AND MATERIAL 
LP. subject 
In order for the subjects to function efficiently, 
they had at their disposal on the grid display console a 
reference chart and a MTACCS Glossary. The reference chart 
contained field header data and allowable field data entries 
used in all MIFASS messages and displays. The MTACCS 
Glossary contained all the abbreviations and acronyms to be 
used. 
2. Experimenter 
Each experimenter had at his disposal the following 
items: a stopwatch, Parametric Data Collection Forms, 
Subject Reaction Forms, the voice recorder, and the input 


message stimuli. 
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Ee. STIMULUS MATERIAL 

All stimulus materials were developed by Hughes Aircraft 
Company in conjunction with Test Bed officers. Message 
input stimuli for the five messages used appear in App. B. 
The formats presented to the subjects for each message 


appear in App. A. 


F. DATA COLLECTION FORMS 
To support the analyses necessary for this test, two 
data collection forms were necessary. 
1. Parametric Data Collection Form 
As depicted in Fig. 2, this form was used to collect 
data for use in determining speed and accuracy measures. 
One form was completed by the applicable experimenter for 
each subject for each format. A total of 32 forms were 
required. 
2. subject Reaction Form 
As discussed earlier, this form was necessary as an 
intermediate step in the nonparametric analysis. A total of 
32 forms were required as each subject completed one for 
each format immediately upon completion of the run testing 


that format. Figure 1 depicts this form. 
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Subject Date 


Format Experimenter 


Message Type Time Now1OL Errors Remarks 


Experimenter Notes 





Figure 2. Parametric Data Collection Form. 
This form was used to gather 
speed and accuracy data for the 
ANOV. 
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Ey. “eeeol PROCEDURE 


A.  PILO@ STUDY 
ye Purpose 

The pilot study was primarily conducted in order to 
validate the test design, determine if the desired data 
could be collected, and see if the required analyses could 
be made using this data. Secondary considerations were 
training of experimenters, refinement of data collection 
procedures and forms, and detection and correction of errors 
in the test stimuli. 

2. Administration 
a. Materials 
The test stimuli, equipment, and data collection 

forms used were those to be used during the actual test. 

bem ee rocedaure 

The test administrator selected those indivi- 

duals to be used as experimenters during the test. Trial 
runs were made with each selectee acting as the subject to 
familiarize him with the entire test. Each selectee then 
performed as an experimenter for other runs using another 
experimenter as a subject. Data was collected as it was to 
be collected during the actual test. This procedure was 
Peploweduuntaetchnes test administrator felt that the level 
of experience was great enough to minimize any effects due 


to experimenter differences. 
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c. Results 
A trial analysis was made using the data 
gathered during the pilot study. The test design was 
confirmed as was the suitability of the data collection 


eas + 


Bee LEST PROCEDURE 
The actual test runs were conducted as follows: 

Prior to each run, the test administrator ensured 
that the system was operating properly and that the correct 
software was loaded. 

The experimenter presented the subject with the 
proper message stimuli and recorded the event on the voice 
recorder. 

The format to be filled in by the subject was called 
to the screen from the memory. 

When the format to be tested appeared, the experi- 
menter recorded the event and started his stopwatch. 

The subject entered the necessary data using the 
alphanumeric keyboard on the grid console. 

Upon completion of his task, the subject pressed 
the STORE RECORD key. The experimenter stopped his stop- 
watch and recorded the time on the voice recorder and the 
Parametric Data Collection Form. 

The above steps were repeated until that run was 
complete. The subject then completed his Subject Reaction 


Form. 
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GeeoaUeukCr BRIEFING 

Prior to commencement of the first test run, a thorough 
briefing was given to all subjects and experimenters. The 
briefing served to acquaint the subjects with the test, its 
purpose, and its procedures. The test administrator pre- 
viewed the test, answered any questions the subjects or 
experimenters had, and provided each subject.with the test 
schedule. The subjects were cautioned not to discuss their 
reactions with other subjects until after completion of the 


entire test. 
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V. ANALYSIS 


A. PARAMETRIC ANALYSIS OF VARIANCE (ANOV) 
1. Procedure 
awe Data Collection 

Test data for the dependent variables speed and 
accuracy were collected on the Parametric Data Collection 
Form. The time required to enter each message stimulus into 
each format by each subject was recorded by the experimenter 
but the accuracy observations were collected differently. 
Errors were detected through the use of a compare program 
by which the computer compared actual entries with the cor- 
rect entries and provided a printout of all entries con- 
taining errors. The Test Administrator then manually 
counted and recorded these errors. The data collected for 
both parametric analyses is presented in App. C. 

bw NoOnpmalizacion Of Data 

In order to preclude effects due to the differ- 
ing lengths of the message types, the test data had to be 
normalized. 

(1) Speed. The normalizing factors for speed 
of data entry were the reciprocals of the number of Keying 
actions necessary to enter a message type into a particular 
format. For example, for Format } Message Type 1, the 
number of keying actions required was 26, so the time for 


each subject to enter the data for this combination was 
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divided by 26 to obtain the normalized times. These times 
appear in App. D. 

(2) Accuracy. The accuracy scores were norm- 
alized in a similar manner; however, the normalizing factors 
were the reciprocals of the number of complete data elements 
to be entered in each format for each message type. These 
normalized observations appear in App. D. 

GreneorgantzZatlonm or Data 

In order to minimize the effort necessary for 
performing the ANOV for both speed and accuracy, the data 
in App. D was rearranged into interaction tables as shown 
ioe. & [1]. 

d. Computations and ANOV Tables 

The computations necessary for obtaining the 
results to be entered in the ANOV tables were accomplished 
using the set of equations summarized in Ref. 1. The 
general form of an ANOV table is discussed in Ref. 1, also; 
therefore, it will not be presented here. 

2. Results 
a. ANOV for Speed of Data Entry 

The ANOV table obtained for the dependent vari- 
able speed in this test is given in Table VII. The effects 
of the different format levels and message types are sig- 
nificant at a = 0.05 as are the interactions of format with 
both message type and subject. The acceptance or rejection 
of the hypotheses concerning speed of data entry iS given 


below: 
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Hoi Rejected 
HO 3 Accepted 
Hos Rejected 
HO Rejected 
Hoo Rejected 
Hod Accepted 
HO13 Accepted 


bee NOV ELOrefoccunracy Of Data Entry 

Results obtained from the ANOV of the dependent 
variable accuracy are displayed in Table VIII. Significant 
effects are exhibited by formats, message types, and their 
mmeeraction at a = 0.05. 

During the test, faulty design in the program 
for Message Type 2 entered in Format 1 required a data 
element to be carried over from one line to the next. The 


margin for the ag 


character on the grid screen was not 
visible so several subjects had difficulty determining 

when to cease entering data on the first line and to con- 
tinue on the second. The higher incidence of errors for 
Format 1 and Message Type 2 made an adjusted ANOV for 
accuracy desirable. Therefore, another ANOV was conducted 
with the errors which were directly attributable to this 
programming problem omitted from the observations. Table IX 
displays the adjusted ANOV results. As shown, format and 
message type effects are no longer significant though 

their interaction remains so. For the two ANOV for accuracy 
of data entry, the acceptance or rejection of the hypotheses 


1Se 
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TABLE VII 


ANOV TABLE FOR SPEED OF DATA ENTRY 
Source Clits Sys MS F 
Replications i -096 = 096 ) 225 
Treatments 
Format (A) 2 PES o8) Pas Te: 9.8357 
Msg. Type (B) 4 As) (3).2 166 Dee ao 
Subject (C) 7 Sa 206 1.244 15.746* 
AB eZ 2.552 IIMS PDAS ENS) 
AC 21 3.6 ue 158 2.00Ce 
BC 28 1 Sal 067 848 
ABC 84 5. 7 oe .069 873 
Bite ts 1 159 2s © lke BOS, 
Roc a 1 Sale 39/7293. 
* 
Significant at a = 0.05. All others were tested against 


a = 0.05 and were not significant. 
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TABLE Vili 


ANOV TABLE FOR ACCURACY OF DATA ENTRY 


Source 


Replications 
Tr atments 
Formats (A) 
Msg. Types (B) 
Subjects (C) 
AB 
AC 
BG 
ABC 


Error 


Total 


* 
Signaticant at 


af 


159 


S19 


lo ye 


SS 


-00006 


All others were 


a = 0.05 and were not significant. 


MS 


- 00006 


SOO eos 
. 00880 
 OOuEy.0 
SOMATA 
-00190 
-00191 
- OOiks2 


JOE 3 


tested against 


OS 


a0 
ei 
soo 
2s 
Oo? 
07 
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TABLE IX 


ADJUSTED ANOV TABLE FOR ACCURACY 
OF DATA ENTRY 


Source aft Ss MS F 
Replications 1 -00026 -00026 me? 
Treatments 

Formats (A) 3 2 OAS ~003G:S 2.44 
Msg. Types (B) 4 POG 24 00206 1 23i6 
Subjects (C) ef -01844 -00264 ar 
AB iz SMI We .00934 6.9 
AC 21 MOO Si -00147 7 
BC 28 VO) SS Site: OORESIS nee: JL 
ABC 84 wel Zoo .00134 89 
Error 13:2) ~-24037 wOOndks L 
(Venera 319 5 Sues) Sy Tl 


* 
Significant at a = 0.05. All others were tested against 
a = 0.05 and were not significant. 
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Original Adjusted 
Ho Rejected Accepted 
Hoa Rejected Accepted 
Hog Accepted Accepted 
Hog Rejected Rejected 
Ho10 Accepted Accepted 
Ho12 Accepted Accepted 
Hoa Accepted Accepted 


Procedure 


NONPARAMETRIC ANALYSIS 


Hatawcollection 


Immediately upon completion of each test run 
each subject completed a Subject Reaction Form on which he 
Marked his evaluation of the ease of use and suitability 
of the format tested. 

b. Analysis 

The Friedman Test is discussed thoroughly by 
Conover and Kirk [Refs. 6 and 7]. It requires two assump- 
tions (which are considered Valad Lor this test) = (1) each 
subject's evaluation of the four formats will be independent 
of every other subject's evaluation, and (2) a subject's 
evaluations may be ranked in some manner. The analysis for 


both ease of use and suitability iS given in App. F. 
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2. Results 
a. Ease of Use 
The test statistic for this analysis was 15.96 
which, when compared with the 0.95 quantile of a Chi Square 
random variable with three degrees of freedom, is 
Slo: £1 Carnes 
Demo cdiod 1aitey 
For this analysis, the test statistic was 11.64 
which is also significant at the 0.05 level. 
c. Hypotheses 
BOeEn Ae 


and Hy were rejected at the 0.05 


HS 16 
level, i.e., the subjects were not indifferent to the 


formats as hypothesized. 
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VI. CONCLUSION 


A. PARAMETRIC ANOV 
1. Speed of Data Entry 

The results of the ANOV for this dependent variable 
clearly show that format variation does affect speed of data 
entry; thus, the primary purpose of this test was success- 
fully accomplished. However, due to significant inter- 
actions of the formats with both message types and subjects, 
interpretation of significant main effects must be qualified. 
Figure 3 illustrates the type of interaction detected between 


formats and subjects. 


Speed 6 Subject 4 


9 Subject 2 


2 3 
Format 


Figure on. Example or a Typical Format=subj ject Interaction. 


It appeared that groups of subjects reacted in similar man- 
ners but that each group reacted differently from the other 
Gacups-meuoreexanple, Ssubygects 1,2;,7, and 8 all reacted to 


Ene wtOrpmats in the Same order, 1.e., Formats 4,3,1l1, and 2 
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(from fastest to slowest), whereas, Subjects 4,5, and 6 
redetea Iimereniewebaen ss 4,2,l, and sen Subject 3 the order 
was 2,3,1,4. No discernible pattern of subject character- 
istics such as I.Q., typing speed, test bed experience, MOS, 
etc., was evident within the groups. Though the subject 
levels were considered fixed, it may be concluded that no 
single format will be best for all the operators in a fielded 
system. For any format chosen, under the present concept, 
some operators will not be able to function optimally. This 
implies that rigid selection and training programs must be 
instituted in order to minimize the effects due to operator 
variations. The format-message type interaction implies 
that some message types are inherently faster (or slower) 
when entered in certain formats. 
2. Accuracy of Data Entry 

The original ANOV clearly rejected the hypothesis 
that neither format nor message type would affect accuracy 
of data entry. However, when ignoring those errors directly 
attributable to the program error, an adjusted ANOV resulted 
in the acceptance of these hypotheses. The results of the 
adjusted ANOV indicate that these factors have little effect 
on accuracy of data entry; however, their interaction is 
Significant. This implies that some of the message types 
are inherently more accurate (or inaccurate) when used with 


Certain tormats. 
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B. NONPARAMETRIC ANALYSIS 

Both hypotheses tested were rejected. The subjects 
tended to consistently rank the formats so that one or more 
were favored over the others. Viewing the totaled ranks for 
each format one may infer that the subjects tended to favor 
Format 3 over the others for both categories. In both cases 


the overall ranking of the formats was 3,4,1, and 2. 


C. SUMMARY 
Peuuepeae Of the Leste 
As discussed in Para. I.D, this test was designed to 
evaluate differences in operator data entry speed and 
accuracy resulting from the use of alternative message for- 
mats when used with differing types of I/O data. The test 
was not designed to determine which format would be best but 
to determine whether or not different formats would affect 
the dependent variables. To this extent, the purposes of 
this test were accomplished. Different formats do affect 
the speed of data entry but do not affect accuracy 
appreciably. 
2. iInferences 
Due to the format-subject interaction, no one format 
can be inferred with certainty to be superior to the others. 
However, the mean speed for Format 3 was the smallest of 
the four formats tested and its error rate was very low. 
It received no rank below two for ease of use and was con- 
Sidered first by four subjects and tied for first by one. 


Concerning operational suitability, Format 3 was ranked 
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Perse Dy shOULTSUbyeECts and tiea@ for first by one. It was 

definitely preferred over the other three formats tested by 
the majority of the subjects even though it was not fastest 
for several. Thus, Format 3 may be inferred, with reserva- 


tions, to be superior for use in a fielded system. 


D. RECOMMENDATIONS 

As shown, the evidence tended to place Format 3 over 
the other three formats for use in the MIFASS. Further 
testing should be contemplated to determine the reasons 
for subject and message type effects. If subject effects 
can be overcome through selection and training of operators, 
then further development can be directed toward determining 
optimum formats for the message types to be used. This 
study can be integrated into the ongoing development of the 
entire system if time does not permit other tests to be 
conducted alone and proper precautions are taken to prevent 
confounding of variables. Another aspect to be approached 
is the weighting of message types with regard to their 
importance to the success of the system. If a particular 
message type is crucial to the success of the system, it 
should have more weight when optimal formats are being 


Gonsildaered,. 
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APPENDIX Bs: MESSAGE INPUT STIMULI 


MeooAGe lver o) 


Field Name Abbreviation Requested mi tri 
Check Fire Msg. CHKFIM; 
Maley PR: AR61 
TO AKO! BC 30 
Target Number TGT NO: BD2121 
Request Number ROQNO: Pozo 0 
Date Time Group DTG: XXXXXXXX 


MESSAGE TYPE 2 


Field Name Abbreviation Reales ted gainers 
End of Mission EOMBDM; 

Battle Damage 
From FR: AA31 
Ike. TOR BA20 
Target Number TGT NO: e120; IL0N6, 
Request Number ROQNO: 2 NOE ONE 
Casualties CAS: 00025 
%$ Destroyed eDES iT. SS, 
Enemy = voseure MY seo de Burn 
Date Time Group DTG: XXXXXXXX 
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MESSAGE TYPE 3 


Field Name Abbreviation Requested Entry 
Target List Input Gaus T ; 
From PR? CA35 
TO TO: AC40 
UTM Location GRID: DE6 4022610 
Altitude PaaT : 00045 
Target Type/Subtype EEYP CEN / BN 
Size SZ: 100X100 
Attitude ATT: 08 
Degree of Protection DOP: HV 
Quantity Ou. 1 
fanget Priority PRI: q 
Target Status STATUS: CONF 
Date Time Group DTG: XXXXXXXX 
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MESSAGE TYPE 4 


Field Name Abbreviation Regus cc Ce bmtery 


Call for Fire by 
Target Number 


From 

To 

Target Number 

Target Type/Subtype 
Quantity 

Size 

Degree of Protection 
Target Mobility 
Attitude 

Method of Engagement 
Weapon Category 
Weapon 

Ordnance 

Fuze 

Drstri bution 

Method of Fire 
Method of Control 


Observer Target 
Darect ion 


Request Number 


Date Time Group 


CFFTGR;} 


Pera. 

EO: 

Gi e NOs 
Mey 
OTY: 

Sl Zur 
DOP : 

TGT MOV: 
/UAY 
METH ENG: 
WCAT: 
WPN: 
ORD: 

Eas 
DISTRB: 
MOF: 
MOC: 


OT DIR: 


RONO: 


DTG: 


ys, 


AG31 

AG20 

Mao 3 2 

VEH /ACFT 
> 

200X200 

MDM 

Slat 

10 


AREA 


018 


QUAIL: 


XXXXXXXX 





MESSAGE TYPE 5 


Field Name Abbreviation Requested Entry 


Fire Order Universal 
From 

To 

Weapon Category 
Target Number 
Request Number 

UTM Coordinates 
Altitude 

Method of Control 
Method of Engagement 
Bestar outvon of Fare 
Time to Open Fire 
Adjusting Unit 


Number of Adjusting 
Rounds 


Adjusting Ordnance 
Adjusting Fuze 
Adjusting Lot 
Adjusting Charge 
Adjusting Angle 


Adjusting Time on 
Target 


Pee erent Titrect. Unit 
FFE No. of Rounds 


FFE Ordnance 


FORDRM; 


FR: 


To: 


WCAT : 


Gay NO: 


RONO: 
GRID: 
ALT : 


MOC : 


METH ENG: 


Dist: 


OPN FIR: 


UADJ: 


RDS: 


ORD: 


EA 
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EC10 
AEOO 
ARTY 
BR2361 
390061 
DE65022500 
00020 
ADJ 
MRK/HA 
ZN 
0600 
G/3/10 


00003 


HE 
Q 
XY 
mI 
26 


0640 


Hy/37 LO 
00004 


ICM 





Field Name Abbreviation Reques ced  emtiay 


PEE er uZc 

FFE Lot 

FFE Charge 

FFE Angle 

FFE Gun Target Line 
eri Unt 

FFE No. of Rounds 
FFE Ordnance 

Bee Fuze 

PEE LOt 

FFE Charge 

FFE Gun Target Line 
ee Uae 

FFE No. of Rounds 
FFE Ordnance 

mee Fuze 

FFE Lot 

FFE Charge 

FFE Angle 

FFE Gun Target Line 


Date Time Group 
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TE 

TW 
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16 
3200 
i737 40 
00006 
SMKRED 
fal 

YX 
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1800 
K/3/10 
00002 
WP 

Vi 
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iz 

24 
4000 
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ACCURACY OF DATA ENTRY 


Fo) 


ERRORS 


Message Type 


= SS 


CO 


1 
2 
3 
4 
5 
6 
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NF Oy] sty wnt wy ft oO 


Format ] 
Format 2? 
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ACCURACY OF DATA ENTRY 


ERRORS 






Message Type 
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ACCURACY OF DATA ENTRY 


ERRORS 


Message Type 


Format 3 


S028 
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APPENDIX E: INTERACTION TABLES 


SPEED OF DATA ENTRY 





2 
SESE e 
259 2.079) 1.8801 25449 ee O24 


2 


ena Kn 
See eee 


55 SNS ale 2) 0,4 PA PAS (2 
1.519 |1.743]2.192|2.398|1.984 
I Bay) 20.501 3 
JEAnaL 3.384 
sea] 1.643] 1.752] 1.990] 2.226 
B18) 2 Se 5018 
2.555] 4#.330]2.766|3.539]3.140 


Aa 


MACH Oncaea lee c Olea cf ie se fo 
O07 Peete. 4032657412. 371 
Smee Looe 2 boon ceo ly 2.174 
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5. 
6 
ei 





Cas 









ee 
a |r.770 eestor e656 [e205 100 1.888)1.463 es O 6 leh. OZ erodes ear 
0 Ol deeroleosezas5 7 O | 5 o4 1.808]1.346 2 ae ele OO ls fal oe 
Pre (SAMS) | es he: || 7A SPs) as: 2.380 ]3.269 1.846|2.694]2.691 pa 


oo 2S) || Ieviowmece sou | 2. 36 2.084]2. 423 Ze OD 2 Lovee easier) 2. 
2 16 | 2. eee 5 6 1a 2 SZ 3 2.684/2.423 2.026 3.242]2.397]2. 


CUl2et 55 2.4202 75642, 881 2.726|2.192 Zive Ow 2.156|2.5432. 
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ACCURACY OF DATA ENTRY 
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AXB 


at 
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B/A 


S00 


- 900 


~143 


~429 


- 498 


eis: 


x Be) 


.249 


~144 


-240 





BXC 
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APPENDIX F: NONPARAMETRIC ANALYSIS 


EASE OF USE 





n Zz. 
2 r5) - 3n(k+1) 


=] i 2 
- Sy iMG £ OO2,93 8 180,95 £ 2eej) = 20g) “ain 
= 15.96% 
X0,05,3 = 7.82 
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X> = B(4) (5) ieicOenz oe 90084 169 +° 306.25] — 3(8) (5) 
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